This is just an example of what preregRS is capable of, not a good-practice example for a preregistration!
I pasted together information from several preregistrations, so the information in this file may not necessarily fit together very well.
This is a Rmd-template for protocols and reporting of systematic reviews and meta-analyses. It synthesizes three sources of standards:
The template is aimed at
We are aware that MARS targets aspects of reporting after the systematic review/meta-analysis is completed rather than decisions and reasoning in the planning phase as PRISMA-P and PROSPERO. MARS nevertheless provides a good framework to determine crucial points for systematic reviews/ meta-analyses to be addressed as early as in the planning phase.
Standards have been partially adapted. Click ‘show changes’ to see changes and reasons for the change.
| Standard | Implemented change | Reason |
|---|---|---|
| MARS | Left out the paper section “Abstract” | An abstract is important for reporting, however not, for planning and registering. |
| MARS | Left out the paper section “Results” and parts of “Discussion” | Specifications on how to report results is important for reporting, however not, for planning and registering. Prospective information on how results will be computed/ synthesized is preserved. |
| MARS | Left out “Protocol: List where the full protocol can be found” | This form serves as a preregistration form, a protocol will be generated later in the research process. |
| MARS | Left out “Give the place where the synthesis is registered and its registry number, if registered” | This template serves as a preregistration form. |
| PROSPERO | Left out non-mandatory fields or integrated them into the mandatory fields. | Avoiding too detailed specifications. All relevant information will be integrated. |
| PROSPERO | Left out some options in “Type and method of review” | Options left out are purely health/ medicine related. |
| PROSPERO | Left out “Health area of the review” | This field is purely health/ medicine related. |
| PRISMA-P | Left out “If registered, provide the name of the registry (such as PROSPERO) and registration number.” | This template serves as a preregistration form. |
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Identify the report as a protocol of a systematic review. If the protocol is for an update of a previous systematic review, identify as such.",
# PROSPERO
"Give the working title of the review, for example the one used for obtaining funding. Ideally the title should state succinctly the interventions or exposures being reviewed and the associated health or social problems. Where appropriate, the title should use the PI(E)COS structure to contain information on the Participants, Intervention (or Exposure) and Comparison groups, the Outcomes to be measured and Study designs to be included. For reviews in languages other than English, this field should be used to enter the title in the language of the review. This will be displayed together with the English language title.",
# MARS
"Title: State the research question and type of research synthesis (e.g., narrative synthesis, meta-analysis)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Identify the report as a protocol of a systematic review. If the protocol is for an update of a previous systematic review, identify as such. |
| PROSPERO | Give the working title of the review, for example the one used for obtaining funding. Ideally the title should state succinctly the interventions or exposures being reviewed and the associated health or social problems. Where appropriate, the title should use the PI(E)COS structure to contain information on the Participants, Intervention (or Exposure) and Comparison groups, the Outcomes to be measured and Study designs to be included. For reviews in languages other than English, this field should be used to enter the title in the language of the review. This will be displayed together with the English language title. |
| MARS | Title: State the research question and type of research synthesis (e.g., narrative synthesis, meta-analysis). |
Example of a preregistration with preregRS
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Type and method of review: Select the type of review and the review method from the lists below. Select the health area(s) of interest for your review.
* Meta-analysis
* Narrative synthesis
* Network meta-analysis
* Review of reviews
* Synthesis of qualitative studies
* Systematic review
* Other",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO |
Type and method of review: Select the type of review and the review method from the lists below. Select the health area(s) of interest for your review.
|
| MARS | Not specified. |
Meta-analysis
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Give the date when the systematic review commenced, or is expected to commence. Give the date by which the review is expected to be completed.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Give the date when the systematic review commenced, or is expected to commence. Give the date by which the review is expected to be completed. |
| MARS | Not specified. |
May 2021 to Oct 2022
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Indicate the stage of progress of the review by ticking the relevant Started and Completed boxes. Additional
information may be added in the free text box provided.
Please note: Reviews that have progressed beyond the point of completing data extraction at the time of
initial registration are not eligible for inclusion in PROSPERO. Should evidence of incorrect status and/or
completion date being supplied at the time of submission come to light, the content of the PROSPERO
record will be removed leaving only the title and named contact details and a statement that inaccuracies in
the stage of the review date had been identified.
This field should be updated when any amendments are made to a published record and on completion and
publication of the review. If this field was pre-populated from the initial screening questions then you are not
able to edit it until the record is published.
* The review has not yet started: [yes/no]
| Review stage | Started | Completed |
|:--------------------------------------------| :----:| :----:|
| Preliminary searches | Yes/No | Yes/No
| Piloting of the study selection process | Yes/No | Yes/No
| Formal screening of search results against eligibility criteria | Yes/No | Yes/No
| Data extraction | Yes/No | Yes/No
| Risk of bias (quality) assessment | Yes/No | Yes/No
| Data analysis | Yes/No | Yes/No
Provide any other relevant information about the stage of the review here (e.g. Funded proposal, protocol not
yet finalised).",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources, escape = F) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PRISMA-P | Not specified. | |||||||||||||||||||||
| PROSPERO |
Indicate the stage of progress of the review by ticking the relevant Started and Completed boxes. Additional information may be added in the free text box provided. Please note: Reviews that have progressed beyond the point of completing data extraction at the time of initial registration are not eligible for inclusion in PROSPERO. Should evidence of incorrect status and/or completion date being supplied at the time of submission come to light, the content of the PROSPERO record will be removed leaving only the title and named contact details and a statement that inaccuracies in the stage of the review date had been identified. This field should be updated when any amendments are made to a published record and on completion and publication of the review. If this field was pre-populated from the initial screening questions then you are not able to edit it until the record is published.
|
|||||||||||||||||||||
| MARS | Not specified. |
| Review stage | Started | Completed |
|---|---|---|
| Preliminary searches | Yes | No |
| Piloting of the study selection process | No | No |
| Formal screening of search results against eligibility criteria | No | No |
| Data extraction | No | No |
| Risk of bias (quality) assessment | No | No |
| Data analysis | No | No |
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Provide name, institutional affiliation, e-mail address of all protocol authors; provide physical mailing address of corresponding author.
* Describe contributions of protocol authors and identify the guarantor of the review.",
# PROSPERO
"
* Named Contact: The named contact acts as the guarantor for the accuracy of the information presented in the register record.
* Named contact email: Give the electronic mail address of the named contact.
* Organisational affiliation of the review: Full title of the organisational affiliations for this review and website address if available. This field may be completed as 'None' if the review is not affiliated to any organisation.
* Review team members and their organisational affiliations: Give the personal details and the organisational affiliations of each member of the review team. Affiliation refers to groups or organisations to which review team members belong.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO |
|
| MARS | Not specified. |
Jürgen Schneider
University of Tübingen
juergen.schneider@uni-tuebingen.de
https://uni-tuebingen.de/de/175743
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Collaborators (name & affilitation) of individuals working on the review, but are not review team member.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Collaborators (name & affilitation) of individuals working on the review, but are not review team member. |
| MARS | Not specified. |
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments.",
# PROSPERO
"Not specified.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments. |
| PROSPERO | Not specified. |
| MARS | Not specified. |
None.
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Indicate sources of financial or other support for the review.
* Provide name for the review funder and/or sponsor.
* Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol.
",
# PROSPERO
"Funding sources/sponsors: Give details of the individuals, organizations, groups or other legal entities who take responsibility for initiating, managing, sponsoring and/or financing the review. Include any unique identification numbers assigned to the review by the individuals or bodies listed. If available, provide grant number(s).",
# MARS
"
* List all sources of monetary and in-kind funding support.
* State the role of funders in conducting the synthesis and deciding to publish the results, if any.
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO | Funding sources/sponsors: Give details of the individuals, organizations, groups or other legal entities who take responsibility for initiating, managing, sponsoring and/or financing the review. Include any unique identification numbers assigned to the review by the individuals or bodies listed. If available, provide grant number(s). |
| MARS |
|
Supported by the Federal Ministry of Education and Research, Germany.
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"List any conditions that could lead to actual or perceived undue influence on judgements concerning the main topic investigated in the review.",
# MARS
"Describe possible conflicts of interest, including financial and other nonfinancial interests."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | List any conditions that could lead to actual or perceived undue influence on judgements concerning the main topic investigated in the review. |
| MARS | Describe possible conflicts of interest, including financial and other nonfinancial interests. |
None.
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe the rationale for the review in the context of what is already known.",
# PROSPERO
"Not specified.",
# MARS
"Problem: State the question or relation(s) under investigation, including
* Historical background, including previous syntheses and meta-analyses related to the topic
* Theoretical, policy, and/or practical issues related to the question or relation(s) of interest
* Populations and settings to which the question or relation(s) is relevant
* Rationale for
(a) choice of study designs,
(b) the selection and coding of outcomes,
(c) the selection and coding potential moderators or mediators of results
* Psychometric characteristics of outcome measures and other variables
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe the rationale for the review in the context of what is already known. |
| PROSPERO | Not specified. |
| MARS |
Problem: State the question or relation(s) under investigation, including
|
Recent research provided evidence that teachers’ professional knowledge regarding the adoption of educational technologies is a central determinant for successful teaching with technologies (Petko, 2012). One prominent conceptualization of teachers’ professional knowledge for teaching with technology is the technological-pedagogical-content-knowledge (TPACK) framework established by Mishra and Koehler (2006). Based on this framework, TPACK encompasses three generic knowledge components (technological knowledge TK, pedagogical knowledge PK, content knowledge CK), three intersections of these knowledge components (technological-pedagogical knowledge TPK, technological-content knowledge TCK, pedagogical-content knowledge PCK) and TPACK as an integrated knowledge component “that represents a class of knowledge that is central to teachers’ work with technology” (p. 1028, Mishra & Koehler, 2006), see Figure 1.
Figure 1. TPACK Model (Mishra & Koehler, 2006; © 2012 by tpack.org)
The most prominent questionnaire to assess TPACK is the self-report questionnaire by Schmidt et al. (2009). This questionnaire encompasses items on the different knowledge components which ask teachers to self-evaluate their confidence to fulfill a task (e.g., an item for TCK is “I know about technologies that I can use for understanding and doing mathematics”, and for TPK is “I can choose technologies that enhance the teaching approaches for a lesson.”). Recently, researchers claim that this self-report questionnaire (and the different extensions and adaptions thereof) rather tap into teachers’ self-efficacy beliefs about teaching with technology than their available knowledge (Scherer et al., 2018; Lachner et al., 2019). Based on the conceptualization of self-efficacy beliefs as the subjective perception of one’s own capability to solve a task (Bandura, 2006), researchers recently argue that the self-report TPACK might be highly intertwined with teachers’ self-efficacy beliefs. Related research suggests that the use of self-report TPACK might be challenging during interpreting the results of empirical studies (Abbitt, 2011; Joo, Kim, & Li, 2018; Fabriz et al., 2020). Therefore, the use of self-report TPACK might induce a jingle-jangle fallacy (Gonzalez et al., 2020; Kelley, 1927). Jingle-jangle fallacies describe a lack of extrinsic convergent validity in two different ways: On the one hand two measures which are labeled the same might represent two conceptually different constructs (jingle fallacy). In the present case, self-report TPACK might differ from teachers’ knowledge for technology-enhanced teaching to a larger extent than previous research suggests. On the other hand two measures which are labeled differently might examine the same construct (jangle fallacy). Accordingly, self-report TPACK and self-efficacy beliefs towards technology-enhanced teaching might be similar constructs with comparable implications on teachers’ technology integration (see e.g., Marsh et al., 2020 for an investigation of jingle-jangle fallacies). However, to date a systematic analysis of the problematic validity of self-reported TPACK is missing.
Against this background, three complementary approaches will be applied in this paper to examine the validity of self-reported TPACK.
First, we will meta-analytically analyze, how the different knowledge components of the TPACK model (i.e., TK, CK, PK, TCK, TPK, PCK) are related to each other across studies when examined with self-report TPACK questionnaires. Measures that depict TPACK components that are more proximal to each other or are intersections of each other in the model should show higher correlations than more distal measures or measure that don’t intersect (e.g., TK should correlate higher with TCK than with PCK).
Second, potential jingle fallacies of self-report TPACK and teachers’ knowledge for technology-enhanced teaching will be examined. Hence, it will be investigated if the two measures represent the same concepts as proposed by researchers (e.g., Schmidt et al., 2009). Therefore, the extent to which self-reported TPACK and more objective measures of teachers’ knowledge for technology-enhanced teaching are related to each other will be examined. If, as proposed, self-report TPACK represents teachers’ knowledge for technology-enhanced teaching, self-report TPACK should be highly related to the quality of technolog use for teaching (see e.g., Kunter et al., 2013; Ericsson, 2006 for the importance of teacher knowledge for generic teaching quality). To investigate the relationship of self-reported TPACK and performance-based measures of teachers’ knowledge for technology-enhanced taching, empirical studies that examine these measures such as studies that investigate the role of self-reported TPACK for the quality of technology-enhanced lesson planning (e.g., Backfisch et al., 2020; Kopcha et al., 2014) or test-based approaches (e.g., Akyuz, 2018; Krauskopf & Forssell, 2013; So & Kim, 2009) will be reviewed.
Third, potential jangle fallacies of self-report TPACK and self-efficacy beliefs towards technology-enhanced teaching will be examined. Therefore, the magnitude of the correlations of self-reported TPACK and self-efficacy beliefs towards technology-enhanced teaching will be compared. Futhermore, the extent to which both measures are related to teachers’ technology integration (e.g., frequency of technology integration) will be analyzed. If self-report TPACK and self-efficacy beliefs are related to the same magnitude to outcome variables, both measures might represent the conceptually similar construct.
ReferencesAbbitt, J. T. (2011). Measuring technological pedagogical content knowledge in preservice teacher education: A review of current methods and instruments. Journal of Research on Technology in Education, 43(4), 281–300. https://doi.org/10.1080/15391523.2011.10782573
Akyuz, D. (2018). Measuring technological pedagogical content knowledge (TPACK) through performance assessment. Computers & Education, 125, 212–225. https://doi.org/10.1016/j.compedu.2018.06.012
Backfisch, I., Lachner, A., Hische, C., Loose, F., & Scheiter, K. (2020). Professional knowledge or motivation? Investigating the role of teachers’ expertise on the quality of technology-enhanced lesson plans. Learning & Instruction, 66, 101300. https://doi.org/10.1016/j.learninstruc.2019.101300
Bandura, A. (2006). Guide for constructing self-efficacy scales. In Self-efficacy beliefs of adolescents (pp. 307–337). https://doi.org/10.1017/CBO9781107415324.004
Ericsson, K. A. (2006). The influence of experience and deliberate practice on the development of superior expert performance. The Cambridge Handbook of Expertise and Expert Performance, 38, 685-705.
Fabriz, S., Hansen, M., Heckmann, C., Mordel, J., Mendzheritskaya, J., Stehle, S., Schulze-Vorberg, L., Ulrich, I., & Horz, H. (2020). How a professional development programme for university teachers impacts their teaching-related self-efficacy, self-concept, and subjective knowledge. Higher Education Research & Development, 1–15.
Gonzalez, O., MacKinnon, D. P., & Muniz, F. B. (2020). Extrinsic Convergent Validity Evidence to Prevent Jingle and Jangle Fallacies. Multivariate Behavioral Research, 1–17.
Joo, Y. J., Park, S., & Lim, E. (2018). Factors influencing preservice teachers’ intention to use technology: TPACK, teacher self-efficacy, and Technology Acceptance Model. Educational Technology and Society, 21(3), 48–59.
Kelley, T. L. (1927). Interpretation of educational measurements.
Kopcha, T. J., Ottenbreit-Leftwich, A., Jung, J., & Baser, D. (2014). Examining the TPACK framework through the convergent and discriminant validity of two measures. Computers & Education, 78, 87–96. https://doi.org/10.1016/j.compedu.2014.05.003
Krauskopf, K., & Forssell, K. (2013). I have TPCK! – What does that mean? Examining the External Validity of TPCK Self-Reports. Proceedings of Society for Information Technology & Teacher Education International Conference 2013, 2190–2197. http://www.stanford.edu/~forssell/papers/SITE2013_TPCK_validity.pdf
Kunter, M., Klusmann, U., Baumert, J., Richter, D., Voss, T., & Hachfeld, A. (2013). Professional competence of teachers: Effects on instructional quality and student development. Journal of Educational Psychology, 105, 805–820. https://doi.org/10.1037/a0032583
Lachner, A., Backfisch, I., & Stürmer, K. (2019). A test-based approach of Modeling and Measuring Technological Pedagogical Knowledge. Computers & Education, 103645. https://doi.org/10.1016/j.compedu.2019.103645
Marsh, H. W., Pekrun, R., Parker, P. D., Murayama, K., Guo, J., Dicke, T., & Arens, A. K. (2019). The murky distinction between self-concept and self-efficacy: Beware of lurking jingle-jangle fallacies. Journal of Educational Psychology, 111(2), 331.
Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for integrating technology in teacher knowledge. Teachers College Record, 108(6), 1017-1054.
Petko, D. (2012). Teachers’ pedagogical beliefs and their use of digital media in classrooms: Sharpening the focus of the “will, skill, tool” model and integrating teachers’ constructivist orientations. Computers & Education, 58, 1351–1359. https://doi.org/10.1016/j.compedu.2011.12.013
Scherer, R., Tondeur, J., & Siddiq, F. (2017). On the quest for validity: Testing the factor structure and measurement invariance of the technology-dimensions in the Technological, Pedagogical, and Content Knowledge (TPACK) model. Computers & Education, 112, 1-17. https://doi.org/10.1016/j.compedu.2017.04.012
Schmidt, D. A., Baran, E., Thompson, A. D., Mishra, P., Koehler, M. J., & Shin, T. S. (2009). Technological pedagogical content knowledge (TPACK) the development and validation of an assessment instrument for preservice teachers. Journal of Research on Technology in Education, 42(2), 123-149. https://doi.org/10.1080/15391523.2009.10782544
So, H. J., & Kim, B. (2009). Learning about problem based learning: Student teachers integrating technology, pedagogy and content knowledge. Australasian Journal of Educational Technology, 25(1), 101–116. https://doi.org/10.14742/ajet.1183
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO)",
# PROSPERO
"State the question(s) to be addressed by the review, clearly and precisely. Review questions may be specific or broad. It may be appropriate to break very broad questions down into a series of related more specific questions. Questions may be framed or refined using PI(E)COS where relevant.",
# MARS
"Objectives: State the hypotheses examined, indicating which were prespecified, including
* Question in terms of relevant participant characteristics (including animal populations), independent variables (experimental manipulations, treatments, or interventions), ruling out of possible confounding variables, dependent variables (outcomes, criterion), and other features of study designs
* Method(s) of synthesis and if meta-analysis was used, the specific methods used to integrate studies (e.g., effect-size metric, averaging method, the model used in homogeneity analysis)"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO) |
| PROSPERO | State the question(s) to be addressed by the review, clearly and precisely. Review questions may be specific or broad. It may be appropriate to break very broad questions down into a series of related more specific questions. Questions may be framed or refined using PI(E)COS where relevant. |
| MARS |
Objectives: State the hypotheses examined, indicating which were prespecified, including
|
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS",
"Added by authors"),
Description = c(# PRISMA-P
"Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review.",
# PROSPERO
"Give details of the types of study (study designs) eligible for inclusion in the review. If there are no restrictions on the types of study design eligible for inclusion, or certain study types are excluded, this should be stated. The preferred format includes details of both inclusion and exclusion criteria.",
# MARS
"Describe the criteria for selecting studies, including
* Independent variables (e.g., experimental manipulations, types of treatments or interventions or predictor variables).
* Dependent variable (e.g., outcomes, in syntheses of clinical research including both potential benefits and potential adverse effects).
* Eligible study designs (e.g., methods of sampling or treatment assignment).
* Handling of multiple reports about the same study or sample, describing which are primary and handling of multiple measures using the same participants.
* Restrictions on study inclusion (e.g., by study age, language, location, or report type).
* Changes to the prespecified inclusion and exclusion criteria, and when these changes were made.
* Handling of reports that did not contain sufficient information to judge eligibility (e.g., lacking information about study design) and reports that did not include sufficient information for analysis (e.g., did not report numerical data about those outcomes).",
# added by authors
"Alternative approaches (to PICO) to describe study characteristics:
* SPIDER: relevant, when including qualitative research (https://doi.org/10.1177/1049732312452938)
* PICOS: Compared to PICO includes study design and reaches higher specifity (ISBN: 978-1-900640-47-3; https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf)
* UTOS: Cronbach's classical framework (ISBN: 978-0875895253)"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review. |
| PROSPERO | Give details of the types of study (study designs) eligible for inclusion in the review. If there are no restrictions on the types of study design eligible for inclusion, or certain study types are excluded, this should be stated. The preferred format includes details of both inclusion and exclusion criteria. |
| MARS |
Describe the criteria for selecting studies, including
|
|
Added by authors |
Alternative approaches (to PICO) to describe study characteristics: SPIDER: relevant, when including qualitative research (https://doi.org/10.1177/1049732312452938) PICOS: Compared to PICO includes study design and reaches higher specifity (ISBN: 978-1-900640-47-3; https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf) UTOS: Cronbach’s classical framework (ISBN: 978-0875895253) |
PICOS
PICOS
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Searches: State the sources that will be searched. Give the search dates, and any restrictions (e.g. language or publication period). Do NOT enter the full search strategy (it may be provided as a link or attachment).",
# MARS
"Describe all information sources:
* Databases searched (e.g., PsycINFO, ClinicalTrials.gov), including dates of coverage (i.e., earliest and latest records included in the search), and software and search platforms used
* Names of specific journals that were searched and the volumes checked
* Explanation of rationale for choosing reference lists if examined (e.g., other relevant articles, previous research
syntheses)
* Documents for which forward (citation) searches were conducted, stating why these documents were chosen
* Number of researchers contacted if study authors or individual researchers were contacted to find studies or to obtain more information about included studies, as well as criteria for making contact (e.g., previous relevant publications), and response rate
* Dates of contact if other direct contact searches were conducted such as contacting corporate sponsors or mailings to distribution lists
* Search strategies in addition to those above and the results of these searches
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Searches: State the sources that will be searched. Give the search dates, and any restrictions (e.g. language or publication period). Do NOT enter the full search strategy (it may be provided as a link or attachment). |
| MARS |
Describe all information sources:
|
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS",
"Added by authors"),
Description = c(# PRISMA-P
"Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated.",
# PROSPERO
"URL to search strategy: Give a link to a published pdf/word document detailing either the search strategy or an example of a search strategy for a specific database if available (including the keywords that will be used in the search strategies), or upload your search strategy. Do NOT provide links to your search results. Alternatively, upload your search strategy to CRD in pdf format. Please note that by doing so you are consenting to the file being made publicly accessible.",
# MARS
"Describe all information sources: Search strategies of electronic searches, such that they could be repeated (e.g., include the search terms used, Boolean connectors, fields searched, explosion of terms).",
# added by authors
"Checklist for Search Strategy: \"PRESS\" Peer Review of Electronic Search Strategies https://doi.org/10.1016/j.jclinepi.2016.01.021"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated. |
| PROSPERO | URL to search strategy: Give a link to a published pdf/word document detailing either the search strategy or an example of a search strategy for a specific database if available (including the keywords that will be used in the search strategies), or upload your search strategy. Do NOT provide links to your search results. Alternatively, upload your search strategy to CRD in pdf format. Please note that by doing so you are consenting to the file being made publicly accessible. |
| MARS | Describe all information sources: Search strategies of electronic searches, such that they could be repeated (e.g., include the search terms used, Boolean connectors, fields searched, explosion of terms). |
| Added by authors | Checklist for Search Strategy: “PRESS” Peer Review of Electronic Search Strategies https://doi.org/10.1016/j.jclinepi.2016.01.021 |
Search String: PICO
NOT (peer OR medic* OR neural-network OR health* OR care*)((computer* OR automat*) AND (writ* AND (argument* OR essay* OR summary OR exposit* OR expla*)) AND (feedback OR evaluat* OR assess* OR scor*)Timespan:
2003 - 2020
Around 2003 several programs were either established of got a major upgrade (see below). Before this timeframe, programs were rarely powered by machine learning (or related) algorythms. They had less or different range of functions and were thus different programs. We therefore choose this timeframe to make make analyses more comparable and reduce bias.
Examples:
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe the mechanism(s) that will be used to manage records and data throughout the review.",
# PROSPERO
"Not specified.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe the mechanism(s) that will be used to manage records and data throughout the review. |
| PROSPERO | Not specified. |
| MARS | Not specified. |
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (that is, screening, eligibility and inclusion in meta-analysis).",
# PROSPERO
"Data extraction (selection and coding): Describe how studies will be selected for inclusion. State what data will be extracted or obtained. State how this will be done and recorded.",
# MARS
"Describe the process for deciding which studies would be included in the syntheses and/or included in the meta-analysis, including
* Document elements (e.g., title, abstract, full text) used to make decisions about inclusion or exclusion from the synthesis at each step of the screening process
* Qualifications (e.g., training, educational or professional status) of those who conducted each step in the study selection process, stating whether each step was conducted by a single person or in duplicate as well as an explanation of how reliability was assessed if one screener was used and how disagreements were resolved if multiple were used."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (that is, screening, eligibility and inclusion in meta-analysis). |
| PROSPERO | Data extraction (selection and coding): Describe how studies will be selected for inclusion. State what data will be extracted or obtained. State how this will be done and recorded. |
| MARS |
Describe the process for deciding which studies would be included in the syntheses and/or included in the meta-analysis, including
|
Two independent reviewers conduct every of the following steps:
Process of revison:
At each of this three steps articles will be selected for inclusion if the inclusion criteria apply and none of the exclusivity criteria apply. If this is not the case articles will be excluded. If it is not clearly wheter articles should be included or excluded these studies will be marked as “maybe” and will be discussed with both reviewer together at the end. Articles will also be discussed together of both of the reviewers if there were disagreements. This disagreements were resolved by screening the full texts using the inclusion criteria.
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators.",
# PROSPERO
"Not specified.",
# MARS
"Describe methods of extracting data from reports, including
* Variables for which data were sought and the variable categories.
* Qualifications of those who conducted each step in the data extraction process, stating whether each step was conducted by a single person or in duplicate and an explanation of how reliability was assessed if one screener was used and how disagreements were resolved if multiple screeners were used as well as whether data coding forms, instructions for completion, and the data (including metadata) are available, stating where they can be found (e.g., public registry, supplemental materials)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators. |
| PROSPERO | Not specified. |
| MARS |
Describe methods of extracting data from reports, including
|
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* List and define all variables for which data will be sought (such as PICO items, funding sources), any pre-planned data assumptions and simplifications
* List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale",
# PROSPERO
"
* Condition or domain being studied: Give a short description of the disease, condition or healthcare domain being studied. This could include health and wellbeing outcomes.
* Participants/population: Give summary criteria for the participants or populations being studied by the review. The preferred format includes details of both inclusion and exclusion criteria.
* Intervention(s), exposure(s): Give full and clear descriptions or definitions of the nature of the interventions or the exposures to be reviewed.
* Comparator(s)/control: Where relevant, give details of the alternatives against which the main subject/topic of the review will be compared (e.g. another intervention or a non-exposed control group). The preferred format includes details of both inclusion and exclusion criteria.
* Main and additional outcome(s): Give the pre-specified main (most important) outcomes of the review, including details of how the outcome is defined and measured and when these measurement are made, if these are part of the review inclusion criteria.
* Measures of effect: Please specify the effect measure(s) for you main outcome(s) e.g. relative risks, odds ratios, risk difference,
and/or 'number needed to treat.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO |
|
| MARS | Not specified. |
Download the Codebook here:
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Not specified.",
# MARS
"Describe the statistical methods for calculating effect sizes, including the metric(s) used (e.g., correlation coefficients, differences in means, risk ratios) and formula(s) used to calculate effect sizes."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Not specified. |
| MARS | Describe the statistical methods for calculating effect sizes, including the metric(s) used (e.g., correlation coefficients, differences in means, risk ratios) and formula(s) used to calculate effect sizes. |
library(metafor)
# fabricated example data to test functionality of code
ex_df <- data.frame(study = c("Author 1 (2020)", "Author 2 (2019)",
"Author 3 (2019)", "Author 4 (2018)"),
m_tc = rnorm(4, 4, 1), # mean treatment group
m_cc = rnorm(4, 3.5, 1), # mean control group
sd_tc = rnorm(4, 1, .1), # SD treatment group
sd_cc = rnorm(4, 1, .1), # SD control group
n_tc = c(22, 25, 44, 32), # n treatment group
n_cc = c(22, 25, 40, 33)) # n control group
# calculate effect sizes as SMD
ex_df <- escalc(measure="SMD", # standardized mean difference
n1i = n_tc, n2i = n_cc, # group sizes
m1i = m_tc, m2i = m_cc, # means
sd1i = sd_tc, sd2i = sd_cc, # standard deviations
data=ex_df)
ex_df## study m_tc m_cc sd_tc sd_cc n_tc n_cc yi
## 1 Author 1 (2020) 2.8593425 2.730474 0.8118327 0.9479845 22 22 0.1434
## 2 Author 2 (2019) 2.5098845 3.720494 0.9732726 0.9613695 25 25 -1.2318
## 3 Author 3 (2019) 4.1319889 4.079114 0.9758181 0.9089996 44 40 0.0555
## 4 Author 4 (2018) 0.9534369 2.858065 0.8929215 0.8140840 32 33 -2.2041
## vi
## 1 0.0911
## 2 0.0952
## 3 0.0477
## 4 0.0989
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS",
"Added by authors"),
Description = c(# PRISMA-P
"Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis.",
# PROSPERO
"Risk of bias (quality) assessment: Describe the method of assessing risk of bias or quality assessment. State which characteristics of the studies will be assessed and any formal risk of bias tools that will be used.",
# MARS
"Describe any methods used to assess risk to internal validity in individual study results, including
* Risks assessed and criteria for concluding risk exists or does not exist.
* Methods for including risk to internal validity in the decisions to synthesize of the data and the interpretation of results.",
# added by authors
"Describe how the quality of original studies are rated. E.g. by 'The Study Design and Implementation Assessment Device (Study DIAD)': https://doi.org/10.1037/1082-989X.13.2.130 "))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis. |
| PROSPERO | Risk of bias (quality) assessment: Describe the method of assessing risk of bias or quality assessment. State which characteristics of the studies will be assessed and any formal risk of bias tools that will be used. |
| MARS |
Describe any methods used to assess risk to internal validity in individual study results, including
|
| Added by authors | Describe how the quality of original studies are rated. E.g. by ‘The Study Design and Implementation Assessment Device (Study DIAD)’: https://doi.org/10.1037/1082-989X.13.2.130 |
We will use a subset of the items/ dimensions from study DIAD (Valentine & Cooper, 2008).
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Describe criteria under which study data will be quantitatively synthesised.
* If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I2, Kendall’s τ).
* If quantitative synthesis is not appropriate, describe the type of summary planned.",
# PROSPERO
"Strategy for data synthesis: Provide details of the planned synthesis including a rationale for the methods selected. This must not be generic text but should be specific to your review and describe how the proposed analysis will be applied to your data.",
# MARS
"Describe narrative and statistical methods used to compare studies. If meta-analysis was conducted, describe the methods used to combine effects across studies and the model used to estimate the heterogeneity of the effects sizes (e.g., a fixed-effect, random-effects model robust variance estimation), including
* Rationale for the method of synthesis.
* Methods for weighting study results.
* Methods to estimate imprecision (e.g., confidence or credibility intervals) both within and between studies.
* Description of all transformations or corrections (e.g., to account for small samples or unequal group numbers) and adjustments (e.g., for clustering, missing data, measurement artifacts, or construct-level relationships) made to the data and justification for these.
* Additional analyses (e.g., subgroup analyses, meta-regression), including whether each analysis was prespecified or post hoc.
* Selection of prior distributions and assessment of model fit if Bayesian analyses were conducted.
* Name and version number of computer programs used for the analysis.
* Statistical code and where it can be found (e.g., a supplement)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
|
PROSPERO |
Strategy for data synthesis: Provide details of the planned synthesis including a rationale for the methods selected. This must not be generic text but should be specific to your review and describe how the proposed analysis will be applied to your data. |
|
MARS |
Describe narrative and statistical methods used to compare studies. If meta-analysis was conducted, describe the methods used to combine effects across studies and the model used to estimate the heterogeneity of the effects sizes (e.g., a fixed-effect, random-effects model robust variance estimation), including Rationale for the method of synthesis. Methods for weighting study results. Methods to estimate imprecision (e.g., confidence or credibility intervals) both within and between studies. Description of all transformations or corrections (e.g., to account for small samples or unequal group numbers) and adjustments (e.g., for clustering, missing data, measurement artifacts, or construct-level relationships) made to the data and justification for these. Additional analyses (e.g., subgroup analyses, meta-regression), including whether each analysis was prespecified or post hoc. Selection of prior distributions and assessment of model fit if Bayesian analyses were conducted. Name and version number of computer programs used for the analysis. Statistical code and where it can be found (e.g., a supplement). |
library(dmetar)
library(tibble)
library(plotly)
# create empty data.frame to be filled
results.power <- data.frame(k = numeric(),
d = numeric(),
power = numeric())
for (k in 5:40) { # loop from k=5 to 40 studies
for (d in c(.25, .30, .35)) {
# compute power analyses for each k (random effects model)
tmp <- power.analysis(d=d, # expected effect size
k=k, # expected number of studies
n1=25, # expected mean group size in treatment group
n2=25, # expected mean group size in control group
p=0.05, # alpha level
heterogeneity = "moderate") # expected heterogeneity
results.power <- add_row(results.power,
k = k,
d = d,
power = tmp$Power)
}
}
# plot results
plot_ly(data = results.power,
x = ~k,
y = ~power,
hovertemplate = "<b>k:</b> %{x} <br /><b>power:</b> %{y}",
type="scatter",
color=~as.factor(d),
mode="lines+markers",
split=~as.factor(d)) %>%
add_lines(x = c(5, 40),
y = c(.8, .8),
inherit = F,
name = "80% power")The results indicate at least 80% power for 9, 12 or 17 included studies under different effect size assumptions. We take on a conservative approach assuming Cohen’s \(d=.25\) and thus will test the synthesized effect size when we are able to include 17 studies.
## k d power
## 1 5 0.25 0.3314842
## 2 5 0.30 0.4464151
## 3 5 0.35 0.5655469
## 4 6 0.25 0.3856321
## 5 6 0.30 0.5157014
## 6 6 0.35 0.6435011
## 7 7 0.25 0.4375575
## 8 7 0.30 0.5790673
## 9 7 0.35 0.7103232
## 10 8 0.25 0.4869204
## 11 8 0.30 0.6362987
## 12 8 0.35 0.7666693
## 13 9 0.25 0.5334934
## 14 9 0.30 0.6874399
## 15 9 0.35 0.8135264
## 16 10 0.25 0.5771421
## 17 10 0.30 0.7327163
## 18 10 0.35 0.8520299
## 19 11 0.25 0.6178080
## 20 11 0.30 0.7724747
## 21 11 0.35 0.8833411
## 22 12 0.25 0.6554939
## 23 12 0.30 0.8071352
## 24 12 0.35 0.9085706
## 25 13 0.25 0.6902505
## 26 13 0.30 0.8371559
## 27 13 0.35 0.9287337
## 28 14 0.25 0.7221656
## 29 14 0.30 0.8630056
## 30 14 0.35 0.9447297
## 31 15 0.25 0.7513544
## 32 15 0.30 0.8851455
## 33 15 0.35 0.9573354
## 34 16 0.25 0.7779518
## 35 16 0.30 0.9040157
## 36 16 0.35 0.9672092
## 37 17 0.25 0.8021056
## 38 17 0.30 0.9200272
## 39 17 0.35 0.9749003
## 40 18 0.25 0.8239713
## 41 18 0.30 0.9335569
## 42 18 0.35 0.9808606
## 43 19 0.25 0.8437078
## 44 19 0.30 0.9449457
## 45 19 0.35 0.9854577
## 46 20 0.25 0.8614735
## 47 20 0.30 0.9544982
## 48 20 0.35 0.9889878
## 49 21 0.25 0.8774243
## 50 21 0.30 0.9624839
## 51 21 0.35 0.9916876
## 52 22 0.25 0.8917112
## 53 22 0.30 0.9691388
## 54 22 0.35 0.9937443
## 55 23 0.25 0.9044786
## 56 23 0.30 0.9746685
## 57 23 0.35 0.9953056
## 58 24 0.25 0.9158639
## 59 24 0.30 0.9792506
## 60 24 0.35 0.9964867
## 61 25 0.25 0.9259959
## 62 25 0.30 0.9830375
## 63 25 0.35 0.9973774
## 64 26 0.25 0.9349955
## 65 26 0.30 0.9861596
## 66 26 0.35 0.9980472
## 67 27 0.25 0.9429745
## 68 27 0.30 0.9887275
## 69 27 0.35 0.9985493
## 70 28 0.25 0.9500365
## 71 28 0.30 0.9908350
## 72 28 0.35 0.9989247
## 73 29 0.25 0.9562765
## 74 29 0.30 0.9925609
## 75 29 0.35 0.9992047
## 76 30 0.25 0.9617814
## 77 30 0.30 0.9939714
## 78 30 0.35 0.9994129
## 79 31 0.25 0.9666306
## 80 31 0.30 0.9951221
## 81 31 0.35 0.9995675
## 82 32 0.25 0.9708959
## 83 32 0.30 0.9960589
## 84 32 0.35 0.9996820
## 85 33 0.25 0.9746425
## 86 33 0.30 0.9968204
## 87 33 0.35 0.9997666
## 88 34 0.25 0.9779291
## 89 34 0.30 0.9974383
## 90 34 0.35 0.9998290
## 91 35 0.25 0.9808085
## 92 35 0.30 0.9979389
## 93 35 0.35 0.9998749
## 94 36 0.25 0.9833280
## 95 36 0.30 0.9983438
## 96 36 0.35 0.9999087
## 97 37 0.25 0.9855300
## 98 37 0.30 0.9986708
## 99 37 0.35 0.9999334
## 100 38 0.25 0.9874524
## 101 38 0.30 0.9989345
## 102 38 0.35 0.9999515
## 103 39 0.25 0.9891288
## 104 39 0.30 0.9991469
## 105 39 0.35 0.9999648
## 106 40 0.25 0.9905891
## 107 40 0.30 0.9993177
## 108 40 0.35 0.9999744
In case we are able to include enough studies we will calculate a random effects model. This is an example of the code with fabricated example data.
# calculating random effects model
rem <- rma(yi, # effect sizes
vi, # variances
data=ex_df)
summary(rem)##
## Random-Effects Model (k = 4; tau^2 estimator: REML)
##
## logLik deviance AIC BIC AICc
## -4.5992 9.1983 13.1983 11.3956 25.1983
##
## tau^2 (estimated amount of total heterogeneity): 1.1699 (SE = 1.0229)
## tau (square root of estimated tau^2 value): 1.0816
## I^2 (total heterogeneity / total variability): 93.65%
## H^2 (total variability / sampling variability): 15.76
##
## Test for Heterogeneity:
## Q(df = 3) = 45.2501, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## -0.7992 0.5596 -1.4280 0.1533 -1.8960 0.2977
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression).",
# PROSPERO
"Analysis of subgroups or subsets: State any planned investigation of ‘subgroups’. Be clear and specific about which type of study or participant will be included in each group or covariate investigated. State the planned analytic approach.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression). |
| PROSPERO | Analysis of subgroups or subsets: State any planned investigation of ‘subgroups’. Be clear and specific about which type of study or participant will be included in each group or covariate investigated. State the planned analytic approach. |
| MARS | Not specified. |
main categorization
categorization for further description
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies)",
# PROSPERO
"Not specified.",
# MARS
"Describe risk of bias across studies, including
* Statement about whether
(a) unpublished studies and unreported data, or
(b) only published data were included in the synthesis and the rationale if only published data were used
* Assessments of the impact of publication bias (e.g., modeling of data censoring, trim-and-fill analysis)
* Results of any statistical analyses looking for selective reporting of results within studies"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies) |
| PROSPERO | Not specified. |
| MARS |
Describe risk of bias across studies, including
|
File-drawer analyses: p-curves for the correlations
Publication bias of single correlations:
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe how the strength of the body of evidence will be assessed (such as [GRADE](https://www.gradeworkinggroup.org/)).",
# PROSPERO
"Not specified.",
# MARS
"Describe the generalizability (external validity) of conclusions, including • Implications for related populations, intervention variations, dependent (outcome) variables."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe how the strength of the body of evidence will be assessed (such as GRADE). |
| PROSPERO | Not specified. |
| MARS | Describe the generalizability (external validity) of conclusions, including • Implications for related populations, intervention variations, dependent (outcome) variables. |
We will examine the strength of our evidence by computing Cohen’s q. Additionally, we will evaluate the overall strength of evidence based on the GRADE framework (Grading of Recommendations Assessment, Development and Evaluation) and examine the following dimensions: overall risk of bias based on publication bias and quality of included studies, inconsistency of findings across studies (if findings across studies are consistent or not), indirectness (if participants of studies are part of the target population).